Adam in Private: Secure and Fast Training of Deep Neural Networks with Adaptive Moment Estimation

Machine Learning (ML) algorithms, especially deep neural networks (DNN), have proven themselves to be extremely useful tools for data analysis, and are increasingly being deployed in systems operating on sensitive data, such as recommendation systems, banking fraud detection, healthcare systems. This underscores the need privacy-preserving ML (PPML) has inspired a line of research into how can constructed efficiently. However, most prior works PPML achieve efficiency by requiring advanced algorithms simplified or substituted with approximated variants that “MPC-friendly” before multi-party computation (MPC) techniques applied obtain A drawback this approach is it requires careful fine-tuning combined MPC might lead less efficient inferior quality (such lower prediction accuracy). an issue secure training DNNs particular, involves several arithmetic thought “MPCunfriendly”, namely, integer division, exponentiation, inversion, square root extraction. In work, we take structurally different propose framework allows evaluation full-fledged state-of-the-art via computation. Specifically, protocols above seemingly MPC-unfriendly computations (but which essential DNN). Our three-party honest-majority setting, both passively actively abort variants. notable feature our they simultaneously provide high accuracy efficiency. enables us efficiently securely compute modern Adam (Adaptive moment estimation) softmax function “as is”, without resorting approximations. As result, DNN outperforms threeparty systems; full up 6.7 times faster than just online phase FALCON (Wagh et al. at PETS’21) 4.2 Dalskov (USENIX’21) standard benchmark network DNNs. The potential advantage even greater when considering more complex realistic networks. To demonstrate this, perform measurements real-world DNNs, AlexNet VGG16, large containing millions parameters. performance these factor 26 ∼ 33 48 51 VGG16 60% 70%, respectively, compared FALCON. Even CRYPTGPU (Tan IEEE S&P’21), optimized runs powerful GPUs, achieves 2.1 4.1 performance,